

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Integration Tests using Teuthology Workflow &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../../" src="../../../../_static/documentation_options.js"></script>
        <script src="../../../../_static/jquery.js"></script>
        <script src="../../../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../../../" id="documentation_options" src="../../../../_static/documentation_options.js"></script>
        <script src="../../../../_static/doctools.js"></script>
        <script src="../../../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../../genindex/" />
    <link rel="search" title="Search" href="../../../../search/" />
    <link rel="next" title="Analyzing and Debugging A Teuthology Job" href="../tests-integration-testing-teuthology-debugging-tips/" />
    <link rel="prev" title="Testing - Integration Tests - Introduction" href="../tests-integration-testing-teuthology-intro/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../../">向 Ceph 贡献：开发者指南</a></li>
          <li class="breadcrumb-item"><a href="../">Teuthology 用户指南</a></li>
      <li class="breadcrumb-item active">Integration Tests using Teuthology Workflow</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../../../_sources/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../architecture/">体系结构</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">开发者指南</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../intro/">简介</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../essentials/">必备知识</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../merging/">何时、合并了什么</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../issue-tracker/">问题追踪器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../basic-workflow/">基本工作流</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../tests-unit-tests/">测试：单元测试</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">测试：集成测试(Teuthology)</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../tests-integration-testing-teuthology-intro/">Introduction</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">Workflow</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#infrastructure">Infrastructure</a></li>
<li class="toctree-l4"><a class="reference internal" href="#getting-binaries-build-ceph">Getting binaries - Build Ceph</a></li>
<li class="toctree-l4"><a class="reference internal" href="#scheduling-test-run">Scheduling Test Run</a></li>
<li class="toctree-l4"><a class="reference internal" href="#viewing-test-results">Viewing Test Results</a></li>
<li class="toctree-l4"><a class="reference internal" href="#killing-tests">Killing Tests</a></li>
<li class="toctree-l4"><a class="reference internal" href="#re-running-tests">Re-running Tests</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../tests-integration-testing-teuthology-debugging-tips/">Debugging Tips</a></li>
<li class="toctree-l3"><a class="reference internal" href="../tests-integration-testing-teuthology-kernel/">Kernel Development</a></li>
<li class="toctree-l3"><a class="reference internal" href="../tests-sentry-developers-guide/">Sentry Notes</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../running-tests-locally/">测试：在本地运行测试</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../tests-windows/">测试: Windows</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../dash-devel/">Ceph Dashboard 开发者文档 (之前是 HACKING.rst)</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../jaegertracing/">Tracing 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../../cephadm/">Cephadm 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../debugging-gdb/">用 GDB 调试</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="integration-tests-using-teuthology-workflow">
<span id="tests-integration-testing-teuthology-workflow"></span><h1>Integration Tests using Teuthology Workflow<a class="headerlink" href="#integration-tests-using-teuthology-workflow" title="Permalink to this heading"></a></h1>
<section id="infrastructure">
<h2>Infrastructure<a class="headerlink" href="#infrastructure" title="Permalink to this heading"></a></h2>
<section id="components">
<h3>Components<a class="headerlink" href="#components" title="Permalink to this heading"></a></h3>
<ol class="arabic simple">
<li><p><a class="reference external" href="https://github.com/ceph/ceph-ci">ceph-ci</a>: Clone of the main Ceph repository, used for triggering Jenkins
Ceph builds for development.</p></li>
<li><p><a class="reference external" href="https://jenkins.ceph.com/">Ceph Jenkins</a>: Responsible for triggering builds, uploading packages
to Chacra, and pushing updates about the build to Shaman.</p></li>
<li><p><a class="reference external" href="https://shaman.ceph.com/builds/ceph/">Shaman</a>: UI Interface used to check build status. In its backend,
it is a REST API to query and store build information.</p></li>
<li><p><a class="reference external" href="https://github.com/ceph/chacra/blob/master/README.rst">Chacra</a>: Service where packages are uploaded. The binaries uploaded
here can be downloaded and used by anyone.</p></li>
<li><p><a class="reference external" href="https://docs.ceph.com/projects/teuthology/en/latest/commands/list.html">Teuthology CLI</a>: Developers can use various Teuthology commands to schedule
and manage test runs.</p></li>
<li><p>Teuthology: This component is responsible for pushing test jobs to
the Beanstalk queue and Paddles. It also picks jobs from
the queue and runs tests.</p></li>
<li><p>Beanstalk queue: A priority queue containing all the queued jobs.
Developers typically do not need to interact with it.</p></li>
<li><p>Paddles: A backend service that stores all test run information.
Developers typically do not need to interact with it.</p></li>
<li><p><a class="reference external" href="http://pulpito.front.sepia.ceph.com/">Pulpito</a>: A UI interface (for information stored in Paddles) that allows
developers to see detailed information about their scheduled tests,
including status and results.</p></li>
<li><p>Testnodes: A cluster of various machines that are used for running tests.
Developers usually schedule tests to run on <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php?id=hardware:smithi">smithi</a> machines, which are
dedicated test nodes for Teuthology integration testing.</p></li>
</ol>
<p>Each Teuthology test <em>run</em> contains multiple test <em>jobs</em>. Each job runs in an
environment isolated from other jobs, on a different collection of test nodes.</p>
</section>
<section id="workflow-overview">
<h3>Workflow Overview<a class="headerlink" href="#workflow-overview" title="Permalink to this heading"></a></h3>
<img alt="../../../../_images/workflow.png" src="../../../../_images/workflow.png" />
<p>To test a change in Ceph, start by pushing a branch with your changes to the
<a class="reference external" href="https://github.com/ceph/ceph-ci">ceph-ci</a> repository. This will automatically trigger the Jenkins process
to build Ceph binaries - the status of the build can be observed on <a class="reference external" href="https://shaman.ceph.com/builds/ceph/">Shaman</a>.
These built packages will be uploaded on <a class="reference external" href="https://github.com/ceph/chacra/blob/master/README.rst">Chacra</a>.</p>
<p>To schedule a Teuthology integration test against this new build, you will
need access to the Sepia lab. Once you have access, log into the Teuthology
machine and complete the one-time initial Teuthology setup required to run
Teuthology commands. After the setup, use the <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command to schedule
a Teuthology run. In this command, use the <code class="docutils literal notranslate"><span class="pre">-c</span> <span class="pre">&lt;ceph-ci</span> <span class="pre">branch</span> <span class="pre">name&gt;</span></code> option to
specify your build. The results of your test can be observed on <a class="reference external" href="http://pulpito.front.sepia.ceph.com/">Pulpito</a>.
Log into a <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php?id=devplayground">developer playground machine</a> to review the Teuthology run’s archive logs.</p>
<p>The rest of the document will explain these steps in detail:</p>
<ol class="arabic simple">
<li><p>Getting binaries - Build Ceph.</p></li>
<li><p>Scheduling Test Run:</p>
<ol class="loweralpha simple">
<li><p>About Test Suites.</p></li>
<li><p>Triggering Teuthology Tests.</p></li>
<li><p>Testing QA changes (without re-building binaries).</p></li>
<li><p>Filtering Tests.</p></li>
</ol>
</li>
<li><p>Viewing Test Results:</p>
<ol class="loweralpha simple">
<li><p>Pulpito Dashboard.</p></li>
<li><p>Teuthology Archives (Reviewing Logs).</p></li>
</ol>
</li>
<li><p>Killing tests.</p></li>
<li><p>Re-running tests.</p></li>
</ol>
</section>
</section>
<section id="getting-binaries-build-ceph">
<h2>Getting binaries - Build Ceph<a class="headerlink" href="#getting-binaries-build-ceph" title="Permalink to this heading"></a></h2>
<p>Ceph binaries must be built for your branch before you can use teuthology to run integration tests on them. Follow these steps to build the Ceph binaries:</p>
<ol class="arabic">
<li><p>Push the branch to the <a class="reference external" href="https://github.com/ceph/ceph-ci">ceph-ci</a> repository. This triggers the process of
building the binaries on the Jenkins CI.</p></li>
<li><p>To ensure that the build process has been initiated, confirm that the branch
name has appeared in the list of “Latest Builds Available” at <a class="reference external" href="https://shaman.ceph.com/builds/ceph/">Shaman</a>.
Soon after you start the build process, the testing infrastructure adds
other, similarly-named builds to the list of “Latest Builds Available”.
The names of these new builds will contain the names of various Linux
distributions of Linux and will be used to test your build against those
Linux distributions.</p></li>
<li><p>Wait for the packages to be built and uploaded to <a class="reference external" href="https://github.com/ceph/chacra/blob/master/README.rst">Chacra</a>, and wait for
the repositories offering the packages to be created. The entries for the
branch names in the list of “Latest Builds Available” on <a class="reference external" href="https://shaman.ceph.com/builds/ceph/">Shaman</a> will turn
green to indicate that the packages have been uploaded to <a class="reference external" href="https://github.com/ceph/chacra/blob/master/README.rst">Chacra</a> and to
indicate that their repositories have been created.  Wait until each entry
is coloured green. This usually takes between two and three hours depending
on the availability of the machines.</p>
<p>The Chacra URL for a particular build can be queried from <a class="reference external" href="https://shaman.ceph.com/api/search/?status=ready&amp;project=ceph">the Chacra site</a>.</p>
</li>
</ol>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The branch to be pushed on ceph-ci can be any branch. The branch does
not have to be a PR branch.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you intend to push master or any other standard branch, check
<a class="reference external" href="https://shaman.ceph.com/builds/ceph/">Shaman</a> beforehand since it might already have completed builds for it.</p>
</div>
<section id="pushing-to-the-ceph-ci-repository">
<h3>Pushing to the ceph-ci repository<a class="headerlink" href="#pushing-to-the-ceph-ci-repository" title="Permalink to this heading"></a></h3>
<p>Follow these steps to push to the ceph-ci repository. After pushing, a new build will
automatically be scheduled.</p>
<ol class="arabic">
<li><p>Add the ceph-ci repository as a remote to your local clone of the Ceph repository:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><style type="text/css">
span.prompt1:before {
  content: "$ ";
}
</style><span class="prompt1">git<span class="w"> </span>remote<span class="w"> </span>add<span class="w"> </span>ceph-ci<span class="w"> </span>git@github.com:ceph/ceph-ci.git</span>
<span class="prompt1"></span>
<span class="prompt1">$<span class="w"> </span>git<span class="w"> </span>remote<span class="w"> </span>-v</span>
<span class="prompt1">origin<span class="w">   </span>git@github.com:ceph/ceph.git<span class="w"> </span><span class="o">(</span>fetch<span class="o">)</span></span>
<span class="prompt1">origin<span class="w">   </span>git@github.com:ceph/ceph.git<span class="w"> </span><span class="o">(</span>push<span class="o">)</span></span>
<span class="prompt1">ceph-ci<span class="w">  </span>git@github.com:ceph/ceph-ci.git<span class="w"> </span><span class="o">(</span>fetch<span class="o">)</span></span>
<span class="prompt1">ceph-ci<span class="w">  </span>git@github.com:ceph/ceph-ci.git<span class="w"> </span><span class="o">(</span>push<span class="o">)</span></span>
</pre></div></div></li>
<li><p>Push your branch upstream by running a command of the following form:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">$<span class="w"> </span>git<span class="w"> </span>push<span class="w"> </span>ceph-ci<span class="w"> </span>wip-yourname-feature-x</span>
</pre></div></div></li>
</ol>
</section>
<section id="naming-the-ceph-ci-branch">
<h3>Naming the ceph-ci branch<a class="headerlink" href="#naming-the-ceph-ci-branch" title="Permalink to this heading"></a></h3>
<p>Prepend your branch with your name before you push it to ceph-ci. For example,
a branch named <code class="docutils literal notranslate"><span class="pre">feature-x</span></code> should be named <code class="docutils literal notranslate"><span class="pre">wip-$yourname-feature-x</span></code>, where
<code class="docutils literal notranslate"><span class="pre">$yourname</span></code> is replaced with your name. Identifying your branch with your
name makes your branch easily findable on Shaman and Pulpito.</p>
<p>If you are using one of the stable branches (<cite>quincy</cite>, <cite>pacific</cite>, etc.), include
the name of that stable branch in your ceph-ci branch name.
For example, the <code class="docutils literal notranslate"><span class="pre">feature-x</span></code> PR branch should be named
<code class="docutils literal notranslate"><span class="pre">wip-feature-x-nautilus</span></code>. <em>This is not just a convention. This ensures that your branch is built in the correct environment.</em></p>
<p>Delete the branch from ceph-ci when you no longer need it. If you are
logged in to GitHub, all your branches on ceph-ci can be found here:
<a class="reference external" href="https://github.com/ceph/ceph-ci/branches">https://github.com/ceph/ceph-ci/branches</a>.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>You can choose to only trigger a CentOS 9.Stream build (excluding other
distro like ubuntu) by adding “centos9-only” at the end of the ceph-ci branch name.
For example, <code class="docutils literal notranslate"><span class="pre">wip-$yourname-feature-centos9-only</span></code>. This helps to get quicker builds
and save resources when you don’t require binaries for other distros.</p>
</div>
</section>
</section>
<section id="scheduling-test-run">
<h2>Scheduling Test Run<a class="headerlink" href="#scheduling-test-run" title="Permalink to this heading"></a></h2>
<section id="about-test-suites">
<h3>About Test Suites<a class="headerlink" href="#about-test-suites" title="Permalink to this heading"></a></h3>
<p>Integration tests are organized into “suites”, which are defined in <code class="docutils literal notranslate"><span class="pre">qa/suites</span></code>
sub-directory of the Ceph repository. These test suites can be run with the teuthology-suite
command.</p>
<p>See <a class="reference external" href="../tests-integration-testing-teuthology-intro/#suites-inventory">Suites Inventory</a> for a list of available suites of integration tests.</p>
<p>More details understanding of how these test suites are defined can be found on <a class="reference external" href="../tests-integration-testing-teuthology-intro/#how-integration-tests-are-defined">Integration Test Introduction Page</a>.</p>
</section>
<section id="triggering-teuthology-tests">
<h3>Triggering Teuthology Tests<a class="headerlink" href="#triggering-teuthology-tests" title="Permalink to this heading"></a></h3>
<p>After you have built Ceph binaries for your branch, you can run tests using
teuthology. This procedure explains how to run tests using teuthology.</p>
<ol class="arabic">
<li><p>Log in to the teuthology machine:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">ssh<span class="w"> </span>&lt;username&gt;@teuthology.front.sepia.ceph.com</span>
</pre></div></div><p>This requires Sepia lab access. To request access to the Sepia lab, see:
<a class="reference external" href="https://ceph.github.io/sepia/adding_users/">https://ceph.github.io/sepia/adding_users/</a>.</p>
</li>
<li><p>For initial setup, follow <a class="reference external" href="https://docs.ceph.com/projects/teuthology/en/latest/INSTALL.html#installation-and-setup">teuthology installation guide</a> to setup teuthology for
your user on teuthology machine. This will enable you to run teuthology commands.</p></li>
<li><p>Run the <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite<span class="w"> </span>-v<span class="w"> </span><span class="se">\</span>
-m<span class="w"> </span>smithi<span class="w"> </span><span class="se">\</span>
-c<span class="w"> </span>wip-devname-feature-x<span class="w"> </span><span class="se">\</span>
-s<span class="w"> </span>fs<span class="w"> </span><span class="se">\</span>
-p<span class="w"> </span><span class="m">110</span><span class="w"> </span><span class="se">\</span>
--filter<span class="w"> </span><span class="s2">&quot;cephfs-shell&quot;</span><span class="w"> </span><span class="se">\</span>
-e<span class="w"> </span>foo@gmail.com</span>
</pre></div></div><p>The options in the above command are defined here:</p>
<blockquote>
<div><table class="docutils align-default">
<thead>
<tr class="row-odd"><th class="head"><p>Option</p></th>
<th class="head"><p>Meaning</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>-v</p></td>
<td><p>verbose</p></td>
</tr>
<tr class="row-odd"><td><p>-m</p></td>
<td><p>machine name</p></td>
</tr>
<tr class="row-even"><td><p>-c</p></td>
<td><p>the name of the branch that was pushed on ceph-ci</p></td>
</tr>
<tr class="row-odd"><td><p>-s</p></td>
<td><p>test-suite name</p></td>
</tr>
<tr class="row-even"><td><p>-p</p></td>
<td><p>the higher the number, the lower the priority of
the job</p></td>
</tr>
<tr class="row-odd"><td><p>--filter</p></td>
<td><p>filter tests in a given suite. The argument
passed to this filter specifies which test you
want to run</p></td>
</tr>
<tr class="row-even"><td><p>-e &lt;email&gt;</p></td>
<td><p>When tests finish or time out, send an email to the
specified address. Can also be specified in
~/.teuthology.yaml as ‘results_email’</p></td>
</tr>
</tbody>
</table>
</div></blockquote>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>The priority number present in the command above is a placeholder.
Do not use it in your own tests. See <a class="reference external" href="../tests-integration-testing-teuthology-intro/#testing-priority">Testing Priority</a> for information
about recommended values.</p>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Do not issue a command without a priority number. The default
value is 1000, a value so large that your job is unlikely ever to run.</p>
</div>
<p>Run <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span> <span class="pre">--help</span></code> to read descriptions of these and other
available options.</p>
</li>
<li><p>Wait for the tests to run. <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> prints a link to
<a class="reference external" href="http://pulpito.front.sepia.ceph.com/">Pulpito</a> where the test results can be viewed.</p></li>
</ol>
<p>The <code class="docutils literal notranslate"><span class="pre">--dry-run</span></code> option allows you to demo-run <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command without
actually scheduling teuthology tests. This is helpful to check how many jobs and which jobs
a command will schedule.</p>
<p>Other frequently used/useful options are <code class="docutils literal notranslate"><span class="pre">-d</span></code> (or <code class="docutils literal notranslate"><span class="pre">--distro</span></code>),
<code class="docutils literal notranslate"><span class="pre">--distro-version</span></code>, <code class="docutils literal notranslate"><span class="pre">--filter-out</span></code>, <code class="docutils literal notranslate"><span class="pre">--timeout</span></code>, <code class="docutils literal notranslate"><span class="pre">flavor</span></code>, <code class="docutils literal notranslate"><span class="pre">--rerun</span></code>,
<code class="docutils literal notranslate"><span class="pre">--limit</span></code> (for limiting number of jobs) , <code class="docutils literal notranslate"><span class="pre">-N</span></code> (for how many times the job will
run), and <code class="docutils literal notranslate"><span class="pre">--subset</span></code> (used to reduce the number of tests that are triggered). Run
<code class="docutils literal notranslate"><span class="pre">teuthology-suite</span> <span class="pre">--help</span></code> to read descriptions of these and other options.</p>
</section>
<section id="testing-qa-changes-without-re-building-binaries">
<span id="teuthology-testing-qa-changes"></span><h3>Testing QA changes (without re-building binaries)<a class="headerlink" href="#testing-qa-changes-without-re-building-binaries" title="Permalink to this heading"></a></h3>
<p>If you are making changes only in the <code class="docutils literal notranslate"><span class="pre">qa/</span></code> directory, you do not have to
rebuild the binaries before you re-run tests. If you make changes only in
<code class="docutils literal notranslate"><span class="pre">qa/</span></code>, you can use the binaries built for the ceph-ci branch to re-run tests.
You just have to make sure to tell the <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command to use a
separate branch for running the tests.</p>
<p>If you made changes only in <code class="docutils literal notranslate"><span class="pre">qa/</span></code>
(<a class="reference external" href="https://github.com/ceph/ceph/tree/master/qa">https://github.com/ceph/ceph/tree/master/qa</a>), you do not need to rebuild the
binaries. You can use existing binaries that are built periodically for master and other stable branches and run your test changes against them.
Your branch with the qa changes can be tested by passing two extra arguments to the <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command: (1) <code class="docutils literal notranslate"><span class="pre">--suite-repo</span></code>, specifying your ceph repo, and (2) <code class="docutils literal notranslate"><span class="pre">--suite-branch</span></code>, specifying your branch name.</p>
<p>For example, if you want to make changes in <code class="docutils literal notranslate"><span class="pre">qa/</span></code> after testing <code class="docutils literal notranslate"><span class="pre">branch-x</span></code>
(for which the ceph-ci branch is <code class="docutils literal notranslate"><span class="pre">wip-username-branch-x</span></code>), run the following
command</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite<span class="w"> </span>-v<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-m<span class="w"> </span>smithi<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-c<span class="w"> </span>wip-username-branch-x<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span>fs<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-p<span class="w"> </span><span class="m">50</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--filter<span class="w"> </span>cephfs-shell</span>
</pre></div></div><p>Then make modifications locally, update the PR branch, and trigger tests from
your PR branch as follows:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite<span class="w"> </span>-v<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-m<span class="w"> </span>smithi<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-c<span class="w"> </span>wip-username-branch-x<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span>fs<span class="w"> </span>-p<span class="w"> </span><span class="m">50</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--filter<span class="w"> </span>cephfs-shell<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--suite-repo<span class="w"> </span>https://github.com/<span class="nv">$username</span>/ceph<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--suite-branch<span class="w"> </span>branch-x</span>
</pre></div></div><p>You can verify that the tests were run using this branch by looking at the
values for the keys <code class="docutils literal notranslate"><span class="pre">suite_branch</span></code>, <code class="docutils literal notranslate"><span class="pre">suite_repo</span></code> and <code class="docutils literal notranslate"><span class="pre">suite_sha1</span></code> in the
job config printed at the beginning of the teuthology job.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you are making changes that are not in the <code class="docutils literal notranslate"><span class="pre">qa/</span></code> directory,
you must follow the standard process of triggering builds, waiting
for the builds to finish, then triggering tests and waiting for
the test results.</p>
</div>
</section>
<section id="filtering-tests">
<h3>Filtering Tests<a class="headerlink" href="#filtering-tests" title="Permalink to this heading"></a></h3>
<p>Test suites includes combinations of many yaml files which can results in massive
amount of jobs being scheduled for a suite. So filters can help to reduce the amount
of jobs or schedule particular jobs within a suite.</p>
<p>Keywords for filtering tests can be found in
<code class="docutils literal notranslate"><span class="pre">qa/suites/&lt;suite-name&gt;/&lt;subsuite-name&gt;/tasks</span></code> in Ceph repository and can be used as arguments
for <code class="docutils literal notranslate"><span class="pre">--filter</span></code>. Each YAML file in that directory can trigger tests; using the
name of the file without its filename extension as an argument to the
<code class="docutils literal notranslate"><span class="pre">--filter</span></code> triggers those tests.</p>
<p>For example, in the command above in the <a class="reference internal" href="#teuthology-testing-qa-changes"><span class="std std-ref">Testing QA Changes</span></a> section, <code class="docutils literal notranslate"><span class="pre">cephfs-shell</span></code> is specified.
This works because there is a file named <code class="docutils literal notranslate"><span class="pre">cephfs-shell.yaml</span></code> in
<code class="docutils literal notranslate"><span class="pre">qa/suites/fs/basic_functional/tasks/</span></code>.</p>
<p>If the filename doesn’t suggest what kind of tests it triggers, search the
contents of the file for the <code class="docutils literal notranslate"><span class="pre">modules</span></code> attribute. For <code class="docutils literal notranslate"><span class="pre">cephfs-shell.yaml</span></code>
the <code class="docutils literal notranslate"><span class="pre">modules</span></code> attribute is <code class="docutils literal notranslate"><span class="pre">tasks.cephfs.test_cephfs_shell</span></code>. This means
that it triggers all tests in <code class="docutils literal notranslate"><span class="pre">qa/tasks/cephfs/test_cephfs_shell.py</span></code>.</p>
<p>Read more about how to <a class="reference external" href="../tests-integration-testing-teuthology-intro/#filtering-tests-by-their-description">Filter Tests by their Description</a>.</p>
</section>
</section>
<section id="viewing-test-results">
<h2>Viewing Test Results<a class="headerlink" href="#viewing-test-results" title="Permalink to this heading"></a></h2>
<section id="pulpito-dashboard">
<h3>Pulpito Dashboard<a class="headerlink" href="#pulpito-dashboard" title="Permalink to this heading"></a></h3>
<p>After the teuthology job is scheduled, the status and results of the test run
can be checked at <a class="reference external" href="https://pulpito.ceph.com/">https://pulpito.ceph.com/</a>.</p>
</section>
<section id="teuthology-archives">
<h3>Teuthology Archives<a class="headerlink" href="#teuthology-archives" title="Permalink to this heading"></a></h3>
<p>After the tests have finished running, the log for the job can be obtained by
clicking on the job ID at the Pulpito run page associated with your tests. It’s
more convenient to download the log and then view it rather than viewing it in
an internet browser since these logs can easily be up to 1 GB in size.
It is also possible to ssh into a <a class="reference external" href="https://wiki.sepia.ceph.com/doku.php?id=devplayground">developer playground machine</a> and access the following path:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">/</span><span class="n">teuthology</span><span class="o">/&lt;</span><span class="n">run</span><span class="o">-</span><span class="n">name</span><span class="o">&gt;/&lt;</span><span class="n">job</span><span class="o">-</span><span class="nb">id</span><span class="o">&gt;/</span><span class="n">teuthology</span><span class="o">.</span><span class="n">log</span>
</pre></div>
</div>
<p>For example: for the above test ID, the path is:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">/</span><span class="n">teuthology</span><span class="o">/</span><span class="n">teuthology</span><span class="o">-</span><span class="mi">2019</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">10_05</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">03</span><span class="o">-</span><span class="n">smoke</span><span class="o">-</span><span class="n">master</span><span class="o">-</span><span class="n">testing</span><span class="o">-</span><span class="n">basic</span><span class="o">-</span><span class="n">smithi</span><span class="o">/</span><span class="mi">4588482</span><span class="o">/</span><span class="n">teuthology</span><span class="o">.</span><span class="n">log</span>
</pre></div>
</div>
<p>This method can be used to view the log more quickly than would be possible through a browser.</p>
<p>To view ceph logs (cephadm, ceph monitors, ceph-mgr, etc) or system logs,
remove <code class="docutils literal notranslate"><span class="pre">teuthology.log</span></code> from the job’s teuthology log url on browser and then navigate
to <code class="docutils literal notranslate"><span class="pre">remote/&lt;machine&gt;/log/</span></code>. System logs can be found at <code class="docutils literal notranslate"><span class="pre">remote/&lt;machine&gt;/syslog/</span></code>.
Similarly, these logs can be found on developer playground machines at
<code class="docutils literal notranslate"><span class="pre">/teuthology/&lt;test-id&gt;/&lt;job-id&gt;/remote/&lt;machine&gt;/</span></code>.</p>
<p>Some other files that are included for debugging purposes:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">unit_test_summary.yaml</span></code>: Provides a summary of all unit test failures.
Generated (optionally) when the <code class="docutils literal notranslate"><span class="pre">unit_test_scan</span></code> configuration option is
used in the job’s YAML file.</p></li>
<li><p><code class="docutils literal notranslate"><span class="pre">valgrind.yaml</span></code>: Summarizes any Valgrind errors that may occur.</p></li>
</ul>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>To access archives more conveniently, <code class="docutils literal notranslate"><span class="pre">/a/</span></code> has been symbolically
linked to <code class="docutils literal notranslate"><span class="pre">/teuthology/</span></code>. For instance, to access the previous
example, we can use something like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">/</span><span class="n">a</span><span class="o">/</span><span class="n">teuthology</span><span class="o">-</span><span class="mi">2019</span><span class="o">-</span><span class="mi">12</span><span class="o">-</span><span class="mi">10_05</span><span class="p">:</span><span class="mi">00</span><span class="p">:</span><span class="mi">03</span><span class="o">-</span><span class="n">smoke</span><span class="o">-</span><span class="n">master</span><span class="o">-</span><span class="n">testing</span><span class="o">-</span><span class="n">basic</span><span class="o">-</span><span class="n">smithi</span><span class="o">/</span><span class="mi">4588482</span><span class="o">/</span><span class="n">teuthology</span><span class="o">.</span><span class="n">log</span>
</pre></div>
</div>
</div>
</section>
</section>
<section id="killing-tests">
<h2>Killing Tests<a class="headerlink" href="#killing-tests" title="Permalink to this heading"></a></h2>
<p><code class="docutils literal notranslate"><span class="pre">teuthology-kill</span></code> can be used to kill jobs that have been running
unexpectedly for several hours, or when developers want to terminate tests
before they complete.</p>
<p>Here is the command that terminates jobs:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-kill<span class="w"> </span>-p<span class="w">  </span>-r<span class="w"> </span>teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi<span class="w"> </span>-m<span class="w"> </span>smithi<span class="w"> </span>-o<span class="w"> </span>scheduled_teuthology@teuthology</span>
</pre></div></div><p>The argument passed to <code class="docutils literal notranslate"><span class="pre">-r</span></code> is run name. It can be found
easily in the link to the Pulpito page for the tests you triggered. For
example, for the above test ID, the link is - <a class="reference external" href="http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/">http://pulpito.front.sepia.ceph.com/teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi/</a></p>
</section>
<section id="re-running-tests">
<h2>Re-running Tests<a class="headerlink" href="#re-running-tests" title="Permalink to this heading"></a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">teuthology-suite</span></code> command has a <code class="docutils literal notranslate"><span class="pre">-r</span></code> (or <code class="docutils literal notranslate"><span class="pre">--rerun</span></code>) option, which
allows you to re-run tests. This is handy when your tests have failed or end
up dead. The <code class="docutils literal notranslate"><span class="pre">--rerun</span></code> option takes the name of a teuthology run as an
argument. Option <code class="docutils literal notranslate"><span class="pre">-R</span></code> (or <code class="docutils literal notranslate"><span class="pre">--rerun-statuses</span></code>) can be passed along with
<code class="docutils literal notranslate"><span class="pre">-r</span></code> to choose which kind of tests should be picked from the run. For
example, you can re-run only those tests from previous run which had ended up
as dead. Following is a practical example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span class="prompt1">teuthology-suite<span class="w"> </span>-v<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-m<span class="w"> </span>smithi<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-c<span class="w"> </span>wip-rishabh-fs-test_cephfs_shell-fix<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-p<span class="w"> </span><span class="m">50</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--r<span class="w"> </span>teuthology-2019-12-10_05:00:03-smoke-master-testing-basic-smithi<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-R<span class="w"> </span>fail,dead,queued<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-e<span class="w"> </span><span class="nv">$CEPH_QA_MAIL</span></span>
</pre></div></div><p>Following’s the definition of new options introduced in this section:</p>
<blockquote>
<div><table class="docutils align-default">
<thead>
<tr class="row-odd"><th class="head"><p>Option</p></th>
<th class="head"><p>Meaning</p></th>
</tr>
</thead>
<tbody>
<tr class="row-even"><td><p>-r, --rerun</p></td>
<td><p>Attempt to reschedule a run, selecting only
those jobs whose status are mentioned by
--rerun-status.</p></td>
</tr>
<tr class="row-odd"><td><p>-R, --rerun-statuses</p></td>
<td><p>A comma-separated list of statuses to be used
with --rerun. Supported statuses: ‘dead’,
‘fail’, ‘pass’, ‘queued’, ‘running’ and
‘waiting’. Default value: ‘fail,dead’</p></td>
</tr>
</tbody>
</table>
</div></blockquote>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../tests-integration-testing-teuthology-intro/" class="btn btn-neutral float-left" title="Testing - Integration Tests - Introduction" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../tests-integration-testing-teuthology-debugging-tips/" class="btn btn-neutral float-right" title="Analyzing and Debugging A Teuthology Job" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>